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Abstract 

Driven by increased safety, efficiency, and 
airspace capacity, automation is playing an 
increasing role in aircraft operations. As aircraft 
become increasingly able to autonomously respond 
to a range of situations with performance surpassing 
human operators, we are compelled to look for new 
methods that help us understand their use and guide 
their design using new forms of automation and 
interaction. We propose a novel design metaphor to 
aid the conceptualization, design, and operation of 
highly-automated aircraft. Design metaphors 
transfer meaning from common experiences to less 
familiar applications or functions. A notable 
example is the “Desktop metaphor” for 

manipulating files on a computer. 

This paper describes a metaphor for highly 
automated vehicles known as the H-metaphor and a 
specific embodiment of the metaphor known as the 
H-mode as applied to aircraft. The fundamentals of 
the H-metaphor are reviewed followed by an 
overview of an exploratory usability study 
investigating human-automation interaction issues 
for a simple H-mode implementation. The 
envisioned application of the H-mode concept to 
aircraft is then described as are two planned 
evaluations. 

Introduction 

Highly automated vehicles, motivated by 
improved performance, efficiency, and safety, are 
becoming commonplace in many forms of 
transportation. As used here, the term “highly 
automated” refers to hardware/software systems 
that can perform a majority of the control functions 
traditionally performed by a human operator. At 


the beginning of the 21st century, the automation of 
state of the art aircraft can control all its actions 
from just after take-off to touchdown and rollout 
during routine operations. 

A large body of experience with highly 
automated airplanes highlights several fundamental 
challenges that limit current system effectiveness 
[1-4]. Briefly summarized, these challenges include 
heavy reliance on memorized action sequences, 
workload spikes caused by reformulation of 
operational tasks into subtasks understood by the 
automation, stupefying workload lulls, 
complacency, skill attrition, inflexible automation 
support, and sudden, unexpected changes in 
automation behavior. Furthermore, unless the basic 
form of human-machine interaction is improved, 
the adverse effects of these challenges are likely to 
be more pronounced as potentially less trained and 
supervised operators ( e.g ., single-pilots or remote 
operators versus a multi-pilot, commercial flight 
crew) use highly automated vehicles. 

The current state of the art is typified by the 
flight management and control systems on 
contemporary transport aircraft (Figure 1). The 
crew can control the vehicle using three primary 



Figure 1. State of the art flight deck 



control automation levels corresponding, in general, 
to manual control, parameter control via an 
autopilot/auto-throttle system, or fully coupled 
operation through a flight management system 
(FMS). The interfaces for the three levels of 
control automation are designed as independent 
systems, resulting in three sets of interfaces (i.e., 
manual control inceptors, mode-control panel, and 
the FMS-Control Display Unit) and system 
characteristics that the operators must be proficient 
with, choose, and manage in any given situation. 
Control of the lateral path, vertical path and speed 
may be simultaneously spread across all three 
interfaces. This situation is complicated by the 
many sub-modes within the primary automation 
levels. These sub-modes are often associated with 
subtle but important changes in system behavior 
( e.g ., whether or not stall protection is provided) 
increasing the likelihood of hazardous operator 
confusion. At the highest automation level (i.e., 
fully coupled operation via the FMS), commands 
are typically entered in the form of abstract, alpha- 
numeric identifiers rather than a spatially encoded 
form (e.g., physically designating features on a map 
or in the external scene). This translation from one 
form to another increases the likelihood of 
accidents such as the crash of American Airlines 
flight 965 near Cali, Columbia [5] In this event, the 
crew mistakenly entered a waypoint unrelated to 
their intended route of flight. Having to look up 
and enter alpha-numeric identifiers can also create 
significant workload, particularly in complex, 
terminal area airspace where workload is often 
already high. 

When utilizing highly automated modes of 
operation, the crew is effectively separated from the 
physical conduct of the flight. While they still have 
critical responsibilities such as monitoring for 
incipient conflicts or hazards, there is no direct 
feedback if these functions are not performed. 
Without regular involvement in the control of the 
vehicle or feedback if more supervisory tasks go 
unperformed, it is possible for crews to fall below a 
safe level of situation awareness, reducing their 
ability to mitigate emerging hazards or respond in 
time-critical situations. In addition, with no 
required physical interaction, functionally 
incapacitated crews may go undetected until the 
situation has already deteriorated to an extreme 


degree. An example of such an occurrence is the 
loss of the Learjet carrying Payne Stewart [6]. 

In addition to these conventional, crew 
controlled functions, safety and security research 
activities are currently developing concepts in 
which the automation may autonomously deviate 
from the crew’s instructions in the presence of a 
significant conflict or hazard [7]. If such 
automation initiated actions occur only during rare, 
uncertain, time-critical, and potentially hazardous 
circumstances, crews are likely to have limited 
proficiency in predicting and verifying the 
correctness of its operation. This lack of familiarity 
increases the likelihood that a crew inappropriately 
disables or overrides the automation or conversely, 
delays a needed intervention. 

Furthermore, traditional automation interfaces 
and associated input parameters (e.g., discrete 
parameter entry via knobs or buttons and alpha- 
numeric identifiers via a keyboard or comparable 
input devices such as voice recognition) are ill- 
suited to the management of spatially referenced, 
autonomous behaviors in dynamic situations. The 
inability to efficiently communicate simple but 
situationally referenced actions may force reversion 
to unaided manual control in many high workload 
situations. While eliminating the possibility of 
human intervention or limiting it severely (e.g., a 
wave-off button) may be seen as solutions to these 
concerns, for complex systems operating in 
uncertain environments, this approach is not 
without numerous and as yet unsolved issues [8, 9]. 
The development of artificial systems with human- 
like “common sense” and creative problem solving 
skills in real-world situations remains a “grand 
goal” as described by leading researchers such as 
Minsky [10]. From a practical perspective, systems 
that do not support flexible human interventions are 
likely to remain more expensive and less robust 
than systems that do for some time to come. 

How then should we apply the ’’lessons 
learned” from late 20th century cockpit automation 
to the design of future aircraft? How do we balance 
between exploiting increasingly powerful 
technologies and retaining authority, with clear 
roles between humans and automation? Is there a 
new way to conceptualize a highly-automated 
vehicle and its operation; perhaps in the form of a 
metaphor? Design metaphors transfer meaning 
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from common experiences to less familiar 
applications or functions. A ubiquitous example is 
the desktop metaphor that has been successfully 
applied to many personal computer operating 
systems for manipulating electronic documents. 
Not all aspects of the original need be copied as a 
metaphor typically has plasticity such that it can, 
and should be, shaped and adapted as necessary to 
support understanding. If the original reference is 
sufficiently intact and familiar to the user, a 
metaphor facilitates easy understanding and 
operation of the new application. Designers and 
engineers usually have to expend a great deal of 
effort in developing and communicating their intent 
not only to users, but also to others involved in the 
development, operation, and training processes. 
This communication can be improved by using a 
‘seed crystal’ such as a suitable metaphor. 

What could be an appropriate metaphor for 
automated vehicles? Let’s leave technology for a 
moment and imagine a situation in everyday life: 

The H-Metaphor: 

Imagine you are riding your bicycle through a 
park with trees and people. You are late for an 
appointment so you’re in a hurry. You’re trying to 
avoid hitting anything and also get to your 



Figure 2. Horse as an autonomous vehicle 
archetype 


appointment on time, but you’re not very familiar 
with this park and you need to keep referring to 
your map. The problem is that it is very difficult to 
steer your bike and read a map at the same time, so 
you are forced to stop. As the time for your 
appointment draws nearer you keep thinking to 
yourself — Is there a way to move through an 
environment with obstacles without having to stop 
every time you need to do something else? How 
about a Horse?! Riding a horse (Figure 2), you 
would be able to read your map and be confident 
that you would not hit any trees or run into people 
because horses instinctively avoid obstacles. Using 
Haptic (manipulative touch, a combination of 
tactile and kinesthetic cues) feedback through the 
seat of your pants and your reins, you are constantly 
aware of what your horse is doing, even while 
focusing your attention elsewhere. If the horse is 
unsure about where to go, it will slow down, and 
seek a new obstacle free path while trying to get 
you back into the loop. The horse might also be 
aware of how engaged you are and adjust its 
behavior. If a dangerous situation suddenly pops 
up, it will try to react before it is too late. You can 
let your horse choose its path without being out-of- 
the-loop or you can take it on tight rein to reassert a 
more direct command. 

Now apply this image to an aircraft. 
Imagine flying through complex airspace with other 
traffic, weather, and tall obstructions while 
comfortably attending to other tasks like navigation, 
communication, or visually searching the 
surrounding terrain. Through the physical feedback 
from your haptic interface, you are constantly aware 
of what your airplane is doing. If your airplane 
senses a conflict, it will take appropriate action and 
you can feel where it’s leading. You can let your 
aircraft choose its path without being completely 
out-of-the-loop, or you can reassert more direct 
control by taking a tighter grip on the stick. The 
airplane would be aware of how engaged you are in 
the control task and adjust its behavior accordingly. 
An extreme example would if the pilot became 
incapacitated. In this event, the airplane would 
declare an emergency and divert to an appropriate 
airport. An H-Metaphor inspired aircraft like this 
could make flying significantly safer and more 
productive. The metaphor is perhaps best suited to 
applications such as single-pilot and Unmanned 
Aircraft System (UAS) operations. These 
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applications have a strong incentive for adopting 
advanced technologies and are not well served by 
simple extrapolations from transport aircraft-like 
interfaces. 


Overview of the H-Metaphor 

The H-metaphor as detailed in Flemisch, et al. 
[11] and summarized here, structures the 
relationship between operator and an intelligent 
vehicle as a partnership between two dissimilar, but 
cooperative agents sharing control of a vehicle (< e.g ., 
Figure 3). This partnership is motivated by the 
emergence of highly automated vehicles capable of 
autonomously performing a wide variety of tasks 
with great precision and vigilance combined with 



Figure 3. Shared control between cooperative 
Agents 


the recognition that their specialized intelligence 
will still be relatively brittle when faced with real- 
world complexities. For example, the ability to 
determine whether a sensed object is a benign 
empty bag or a hazardous chunk of concrete could 
be important in a variety of situations but is far 
from feasible in any general sense. The H- 
metaphor addresses this situation by relying on the 
general intelligence of a human operator. In the 
partnership, the human strategizes, corroborates, 
and backs up actions of the automation in the 
accomplishment of operations that are beyond the 
vehicle’s reliable, independent cognition. The 
human is aided in this role by the vehicle’s 
awareness of the routine situation elements; ability 


to perform immediate trajectory control tasks; the 
intelligence of its interaction; and its ability to 
independently monitor and maintain immediate 
operational safety. The goals of this coupling are 
improved overall system safety, performance, and 
flexibility beyond what could be accomplished with 
conventional automation or manual control while 
significantly reducing the cost and complexity of 
the needed automation and operator training. 
Furthermore, the metaphor provides a path for the 
incremental introduction of increasingly capable 
automation technologies and may be a necessary 
evolutionary step toward, and in many cases an 
alternative to, autonomous vehicles that routinely 
operate in complex, safety-critical environments 
without human backup. 


The metaphor provides guidance on the 
human-vehicle interaction, the vehicle as an 
autonomous agent, and multi-vehicle interaction 
[11]. A horse and rider system interacts through 
multi-modal communication underpinned by a 
strong, bi-directional haptic component. Human 
haptic senses (e.g., proprioception and cutaneous 
touch) are unique in that they both sense and effect 
the external environment. Furthermore, as a vector 
quantity (i.e., force) they provide a natural means of 
communicating spatial concepts such as the 
direction of an intended or desired maneuver. 
These properties enable robust and rapid, bi- 
directional communication and negotiation between 
horse and rider. For example, even in the case of a 
horse cart without direct bodily contact, the reins 
form a bi-directional link with the horse able to 
transmit information to the driver as well as receive 
it. According to the German National Equestrian 
Federation, turns are performed by yielding the 
outside rein with a twist of the hand(s), not pulling 
the inside one (Figure 4). There should always be a 
’’soft, steady, elastic connection” between the 
driver’s hand and horse’s mouth, where ’’the horse 



Figure 4. Hands and reins position for elastic 
connection [12] 
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seeks the contact and the driver provides it" [12]. If 
the horse wants to change direction, perhaps to 
avoid a rut, it leads with a turn of its head. The 
rider feels a change in rein tension and can accept 
or reject the action by relaxing or increasing the 
tension. In addition, discrete communication 
elements such as an agitated jerk (possibly 
accompanied with an auditory snort and visual tail 
flick) are often combined with such continuous 
signals. Furthermore, from a lack of any tension 
the horse can infer that the driver is relatively 
disengaged and adjust its expectations accordingly. 

Efficient bi-directional communication results 
in a type of shared control not present in 
conventional automation. Conventional automation 
tends to be 4 all or nothing’ where either the human 
or the machine has control. In comparison, the 
rider of a horse can hold the locus of control while 
the horse remains “in-the-loop” performing inner- 
loop functions and monitoring for hazards (i. e., 
tight reins operation). Alternatively, the rider can 
also explicitly or implicitly designate a clear task or 
behavior of some duration for the horse to perform 
and allow the horse to have the locus of control 
(i e.g ., loose reins). The rider, even in loose reins, 
remains in-the-loop and can flexibly control 
important aspects of the behavior such as what 
position to maintain relative to another, running 
animal (e.g., calf roping ) or how to avoid a hazard 
(e.g., left, right or over). 

It should be recognized that the H-metaphor 
applies to the near-term or tactical actions of the 
vehicle. Horses, outside of TV and movies, do not 
worry about what is hours ahead relative to the 
goals of the rider. As part of a complete system 
concept, a separate, but complementary planning 
sub-system is envisioned to support longer term, 
mission planning and monitoring requirements 
(e.g., Schutte and Goodrich [13]). The maximum 
look-ahead of an H-inspired system is at the 
discretion of the designers, but in keeping with the 
metaphor, the human-automation interaction in 
particular, would nominally project only as far as 
needed to support the current action and possible 
transitions while accounting for local hazards. 
While it may seem counterintuitive to limit the 
ability to pre-program sequences of action as far 
into the future as technologically feasible, the intent 
of the limitation is two fold. The first is to ensure 


that the operator has regular, but relaxed, physical 
involvement throughout a mission, thus aiding 
maintenance of situation awareness. The second 
motivating factor is optimizing the interaction for 
robust communication of near-term, possibly time 
and safety critical goals and actions. This near-term 
focus serves to minimize miscommunication and/or 
protracted exchanges by keeping the language set 
and range of situationally applicable behaviors 
small and with a minimum of abstraction. 

The H-mode, a Realization 

The metaphor provides a broad vision as to 
how an H-inspired vehicle should interact and 
behave. Achieving this vision requires better 
understanding the fundamental human-automation 
interaction issues, applying and advancing the 
underlying technologies for autonomous systems, 
and integrating this knowledge into practical 
designs for specific operational applications such as 
ground vehicles, aircraft and spacecraft. Ideally, 
this work should result in specific, yet malleable 
automation design and user interaction guidelines 
that could be applied across a wide range of 
“intelligent” vehicles. By facilitating positive 
transfer of training, knowledge, and experience 
between operational domains and reducing the 
potential for confusion, we could expect significant 
improvements in safety, operational effectiveness 
and training productivity for both users and 
developers. 

The H-mode, a practical realization of the 
metaphor using commercially available haptic 
interface hardware (e.g., an active, force-feedback 
side-stick) and with applicability across operational 
domains, is currently being developed by NASA 
and the German Aerospace Lab, DLR. Early steps 
in this process include an exploratory study 
conducted first at NASA and latter extended at the 
DLR. As described by Flemisch, Goodrich, and 
Conway [14], this study investigated the general 
user acceptance and expectations of the metaphor 
and developed an initial, generic “language” for the 
interaction between a simple automated vehicle and 
a human operator. A small but broadly 
representative group of prototypical users 
participated in a structured interview to explore 
their “naive” reaction to the metaphor and 
expectations for, and acceptance of, an intelligent 
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H-inspired vehicle. The results of these interviews 
(the questionnaire and responses are reproduced in 
ref. 14), indicate overall acceptance of the metaphor 
and that the general issues and capabilities of such 
vehicles are intuitively understood. 

The interaction investigation consisted of a 
“structured exploration” to develop and evaluate a 
preliminary interaction language between a simple 
automated vehicle simulation and a human operator 
using an active side-stick for the following generic 
1 -dimensional control tasks: 

• Starting and Stopping the vehicle 

• Maintaining a desired spacing interval behind a 
lead vehicle. As shown in figure 5, deviations 
from the desired interval were indicated by 
triangular zones of increasing risk and 
rectangular zones of significant danger. The 
subjects and “H” did not necessarily have the 
same zones, requiring a haptic negotiation to 
find the best compromise. 

• Transitioning between a more automated and a 
less automated mode of operation (Tight 
Rein/Loose Rein) 

The operator performed the last two tasks in 
parallel with a secondary task to mimic the divided 
attention likely in actual operations. 

A goal of the exploration was to match the 
expectations of users ( e.g “What behavior would 
you expect of this artifact in that situation?”) with 
the technical potential of the design space and the 
ideas of the designer. One of the tools used to 
achieve this match was a “theater” system or 
“Wizard of Oz” technique, based on the concept of 
Salber and Coutaz [14]. A confederate “behind the 
curtain” simulated parts of the automation via a 
second side-stick that was electronically linked to 

Situation “Ice”, with differing goals 


participant confederate/automation 



Figure 5. Distance task from ref. 14 



Figure 6. Theater system with curtain 
open 

the subject’s stick. The technique allowed flexible 
exploration of users expectations and possible 
design variations prior to interactions with a rapid 
prototype implemented in technology. While in the 
original technique described in ref. 15, the 
participant does not know about the confederate, 
here the participant knew about it from the very 
beginning. The “curtain” was open in the training 
phases (Figure 6) and closed during the usability 
assessments. 

The exploration went through the following 
stages: 

• Pre-Design of potential interaction patterns by a 
core design team in the theater. 

• Usability assessment and refinement of the pre- 
design with other team members. 

• Rapid prototyping implementation of selected 
features of the pre-design. 

• Structured interview with prototypical users. 

• Laboratory based assessment with prototypical 
users. 

o Collection of expectations directly in the 
theater system 

o “Naive runs” (users interact with Pre-design 
without any training) 

o Teaching of the pre-design and comparison 
to expectations 

o Usability assessment of pre-design with the 
theater system 

o Usability assessment of pre-design with the 
rapid prototypes 
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o De-Briefing 

• Harmonization of the pre-design with the 
expectations 

While a detailed description of the results goes 
beyond the scope of this paper, a first draft of 
interaction consists of the following haptic 
communication cornerstones: 

- Vibrations: A specific scheme of vibrations that 
combines information about danger and degree of 
automation (“Alive”-vibration), combined with 

- Pushing: A specific scheme of pushing back the 
stick from danger, combined with 

- Signaling: Discrete interaction “double-ticks” for 
signaling the desire for discrete distance or speed 
changes, combined with 

- Transitioning: A specific scheme of fluidly 
transitioning between a higher level of 
automation (Loose Rein) and a lower level of 
automation (Tight Rein) 

o Tight Rein -> Loose Rein by first leading the 
H-vehicle into the desired maneuver and then 
(Variant 1) relaxing the hand on the stick or 
(Variant 2) a discrete “yes”-signal or (Variant 
3) a combination of 1 and 2. 

o Loose Rein -> Tight Rein by taking a firm 
grip, if necessary combined with a discrete 
“no”-signal. 

- Fine Tuning: A specific scheme of fine tuning 
reference values the automation in Loose Rein 

- Arbitration: Specific schemes of prioritizing 
between the will of the user and of the automation 

While a couple of additional design ideas 
surfaced and a few glitches were found during the 
usability assessment, overall the pre-design 
matched most of the expectations of the evaluation 
subjects and the prototypes were, for this state of 
development, quite intuitive. 

Toward a Comprehensive Aircraft 
Implementation 

DLR and NASA are currently investigating 
more comprehensive applications of the H-mode to 
ground and air vehicles, respectively. The current 
status of the ground vehicle application is 


summarized in Flemisch, Schieben, and Schindler 
[16] and the aviation application is described here. 

Technical Perspective 

Figure 7 illustrates the conceptual system 
architecture. Major system elements include the 
multi-modal user interface including an active or 
haptic side-stick and speed-command lever (not 
shown); inner-loop control augmentation; real-time 
sensors and/or data link system; and the H-inspired 
automation (HI A). 

Within the user interface, the active stick 
and speed-command lever provide a common 
means by which both the operator and HIA issue 
commands to the inner-loop. The haptic interaction 
allows each agent to sense the others involvement 
( e.g. , applied force) and infer an associated intent. 
Visual displays (i.e., a primary flight and tactical 
map displays) provide graphical representations of 
the external environment overlaid by basic flight 
parameters and the HIA’s interpretation of this 
information. Since the inner-loop automation 
provides full-time stability and control 
augmentation, presentation of basic flight 
parameters is primarily for awareness rather than 
control and can be significantly simpler than current 
formats. The displayed environmental information 
includes physical elements such as terrain, 
obstructions, weather, and traffic and virtual 
elements such as airspace boundaries, published 
fixes and procedures. The route from a mission 
planning system, if present, is also included. 
Elements rendered on the displays are also known 
to the HIA. HIA specific symbology allows the 
pilot to quickly ascertain its status, motivation, 
center of attention, projected trajectory, and how it 
would respond to potential directives. Finally, 
although not currently implemented in a prototype, 
auditory interface features will be used to 
compliment the haptic and visual elements. In 
keeping with the H-metaphor, these 
communications can be bi-directional but limited in 
content and complexity {e.g., not complex 
sentences or abstractions). 

The output of the stick provides the reference 
signal acted on by the inner-loop control element, 
essentially a fly-by-wire autopilot providing flight- 
path oriented, control-wheel steering and envelope 
protection (e.g., [17-20]). Considering the horse 
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Figure 7. H-mode based flight control system 


metaphor, the inner-loop can be likened to the hind 
brain, spinal cord, and voluntary neuromuscular 
system. The inner-loop simplifies the pilot-vehicle 
interaction into an easily managed set of elemental 
actions. This simplification is accomplished by 
delegating management of a number of high- 
frequency, non-linear, but largely deterministic 
control processes to the inner-loop. This 
aggregation also reduces the workload and training 
of the operator and simplifies the design and 
understanding of the outer-loop automation 
elements. Obviously, the effects of possible failure 
modes and other “non-normals” need to be 
considered on overall system behavior. The current 
state of the art is rapidly enabling retention of 
augmented response characteristics in the presence 
of component failures, albeit with some 
performance degradation, rather than reversion to a 
completely different set of dynamics ( e.g ., a direct 
electric link mode). Optional, advanced functions 
within the inner-loop include upset recovery and 
adaptive processes compensating for vehicle 
changes caused by airframe damage or other effects 
(e.g., airframe icing) such as described by Burken, 
et al [21]. For maximum capability, these functions 
should communicate operating envelope changes to 


other system elements (i.e., HAI, interaction 
manager) so that they can appropriately update their 
internal models. 

The sensor and data link capabilities allow the 
vehicle to obtain real-time information about the 
local environment, functioning as the external 
senses of the metaphor. These sensors could 
include conventional instruments such as satellite 
based navigation (e.g., GPS), weather radar, etc, as 
well has emerging technologies like multi-spectral 
machine vision for navigation and hazard avoidance 
and digital data link based state and intent sharing 
between vehicles and air traffic control. One of the 
benefits of the H-mode concept is that new sensor 
capabilities can be readily integrated into its 
framework rather than being added in a stand alone 
fashion with “integration” largely left to the pilot as 
is often done today. 

Within the HI A, relevant static information 
stored in databases is analogous to long term 
memory. This static data provides the HIA with 
knowledge of nonphysical elements such as 
airways, static physical features such as terrain, and 
databases characterizing its own capabilities. 
Within the situation assessment function, real-time 
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and static data are analyzed, features identified, and 
an integrated representation of the current situation 
constructed. The situation assessment also includes 
relevant inputs from the operator. The important 
aspects of the perceived situation are communicated 
back to the operator via the interaction manager. 
When operating in loose reins mode, the HIA 
creates and maintains a desired trajectory which is 
followed by a guidance subsystem. In tight reins 
mode, the current aircraft implementation does not 
generate a reference trajectory and no maneuvering 
command is sent to the stick unless an imminent 
conflict is perceived. Alternatively, the DLR is 
investigating the soft haptic presentation of a 
desired trajectory even during tight reins operation. 
This presentation may assist the operator and 
increase their awareness of potential future actions 
by the HIA and be particularly useful in situations 
where the scope of safe tight reins maneuvering is 
limited. 

Various means of conflict avoidance are under 
investigation at both organizations. Currently, 
simple reactive behaviors such as potential-field 
guidance [22] appear to be an intuitive, robust and 
predictable means of performing obstacle 
avoidance. In addition, superposition of individual 
guidance commands ( e.g ., path following + 
avoidance) aided by simple heuristics, is being 
investigated as a means of effectively reacting to 
simultaneous influences. 

Pilot’s Perspective 

The H-mode system integrates control of the 
vehicle’s full range of control automation into a 
single consistent interface system and natural set of 
interaction skills. The pilot uses the same basic 
control and display devices for augmented manual 
control through complex behaviors such as takeoffs, 
landings, coupled procedures, and automated 
conflict/hazard avoidance. 

The H-mode emphasizes two basic modes of 
operation corresponding to tight and loose reins. In 
the tight reins mode, the pilot holds the locus of 
immediate control while in loose reins the aircraft 
autoflight system has a relatively high degree of 
autonomy in satisfying longer-term directives. 
Transitions between tight and loose are managed by 
an “arbitration function” in the interaction manager. 
Other modes of operation supporting more 


extensive autonomous actions are possible design 
extensions but are not discussed here as they are 
perhaps best viewed outside the context of the 
metaphor. 

In tight reins, the pilot commands the near- 
term actions of the vehicle through the inner-loop 
control and does not communicate any sustained 
intent to the HIA. In this mode, the HIA provides 
no or limited aid unless it perceives an emerging 
conflict or hazard. In the event of a perceived 
conflict, the HIA brings the situation to the pilot’s 
attention via the visual and haptic interface 
elements. If the conflict severity exceeds a defined 
threshold, the automation sends a force input, 
effectively a maneuver command, to the stick. If 
unopposed, this input safely redirects the vehicle. 
The pilot can also perform their own deconfliction 
maneuver or, if for some reason they need to 
continue toward the conflict, apply an appropriate 
counter-force. The degree and duration of the 
automation’s resistance can be situationlly tailored 
by the arbitration function. For example, once the 
pilot has demonstrated an unambiguous and 
informed intent to continue toward a hazard, it may 
be advisable for the automation to reduce or cease 
its opposition. Such situations might arise due to 
erroneous or incomplete local situation awareness 
and/or knowledge of the mission objective by the 
HIA. Limiting the strength and/or duration of 
opposition provides a mechanizm through which 
the pilot retains authority. 

In loose reins, the HIA performs a well defined 
task or behavior. This behavior may be initiated 
through explicit direction from the pilot or implicit 
in the local situation {e.g., flare before touchdown). 
Flying published procedures or a preplanned route 
are examples of actions normally performed in 
loose rein. Several behaviors may be combined 
such as route following while maintaining spacing 
from other traffic. Other loose rein behaviors 
include precise, high-bandwidth tasks such as 
takeoff and flare and landing. 

In loose reins, the pilot is relieved from 
continuously generating instantaneous commands 
but remains actively involved in managing and 
transitioning between behaviors. By lightly 
touching the haptic interface elements, the pilot can 
feel the HIA’s instantaneous commands, providing 
a non-visual means of maintaining awareness. This 
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awareness is aided by discrete haptic signals 
indicating the general situation status and imminent 
events such as reaching an intersection or planned 
trajectory transition ( e.g ., top of descent). 

The pilot also uses the haptic interface to 
modify key parameters within a behavior and 
manage transitions between behaviors. Within a 
behavior, pilot applied forces can be fed back to the 
HIA and used to update its active reference targets 
in an operationally intuitive manner. For example, 
in a following task, pressure on the speed command 
lever is fed back to adjust the separation interval. 
Similarly, in a landing behavior, light stick forces 
can be used to adjust the aim point used in the 
landing guidance. The HIA’s receptiveness to 
these updates is situationally tailored such that only 
safe and appropriate actions are supported in loose 
reins. The pilot’s awareness of this receptiveness is 
supported via multi-modal cues including tailoring 
of the basic feel characteristics of the haptic 
interface (e.g., a heavier feel as limits are 
approached). 

At major transition or decision points, the HIA 
can be configured to react only after the pilot has 
made a positive control input indicating they are 
aware of the situation. The pilot typically makes 
this input by manually initiating the transition. This 
interaction helps prevent the detachment and loss of 
situation awareness that can occur with 
conventional automation and encourages the pilot to 
verify the appropriateness of major transitions. If 
the pilot fails to initiate an expected transition, the 
aircraft provides increasingly salient cues. If the 
pilot still does not respond, the automation shifts 
toward a more conservative (i.e., safer) posture or 
action. At decision height, for example, the pilot is 
expected to make an input confirming that it is safe 
to proceed and transition from the approach to the 
landing behavior. If this input is not sensed, the 
aircraft initiates a missed approach. The pilot can 
nominally alter or reject an expected transition by 
making a light control input consistent with an 
alternate loose-reins action or by making a firm 
input, triggering a transition to tight reins. 

If the HIA perceives an emerging conflict 
during loose reins operations, it brings the situation 
to the operator’s attention. If the HIA is capable of 
generating a safe, straightforward deconfliction 
maneuver, it will be offered as an option to the 


pilot. Like a normal decision point, the pilot can 
initiate the action with a simple, positive control 
input while remaining in loose reins. If not satisfied 
with any of the loose reins resolution actions, the 
pilot has the option of transitioning into tight reins. 
If the pilot does not respond, or if there is 
insufficient time to allow reaction, the HIA 
autonomously shifts toward a conservative posture. 

Transitions between loose and tight reins are 
managed by an arbitration function in the human- 
machine interaction manager. The interaction 
manager interprets the pilot’s inputs in the context 
of the HIA’s perceived situation. This 
interpretation is aided by a small set of discrete 
elements allowing simple, but direct 
communication, similar to the interaction 
commonly seen between a human and a well trained 
animal. These elements include signals meaning 
‘yes’, ‘no’, ‘NO’, ‘I am unsure’, and ‘I want your 
attention’. The interaction manager typically 
encodes these multi-modally including a haptic 
component. Current research is exploring a variety 
of generation methods from the pilot’s side, 
including haptic signals and dedicated buttons. 

In general, if the pilot sustains a firm input 
while in loose reins, a transition into tight reins 
occurs. If, while operating in tight reins, the pilot 
maintains the vehicle in a specified relationship 
relative to certain external elements, the automation 
offers or initiates the appropriate loose reins 
behavior. Figure 8, for example, shows the pilot 
directing the automation to enter a published 
holding pattern. First, the pilot maneuvers the 
airplane so that the flight path marker (FPM) 
symbol is stabilized on a visualization of the fix 
associated with the hold. Rendered as a transparent 
“pole”, the visualization allows the pilot to 
graphically specify the altitude of the procedure. 
The vertical extent of the pole indicates the range of 
appropriate altitudes and the projection of the FPM 
on the pole, rounded to the nearest appropriate 
increment, specifies the desired altitude. As the 
pilot stabilizes on the pole, the HIA highlights the 
procedure, indicating that it can transition to loose 
reins. The pilot completes the transition via a “yes” 
signal and a magenta line provides a visual 
indication of the trajectory guiding the automation. 
Simultaneously, the basic feel characteristics of the 
stick are changed to indicate the vehicle has the 
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Figure 8. Transition from tight to loose riens 


locus of control. In some situations, several loose 
reins behaviors may be applicable and additional 
discrete communication is used to select or clarify 
the desired action. Situations in which the HIA 
requests a transition from loose to tight will be 
encountered as it becomes unsure of the situation. 
Depending on the situation, the operator may or 
may not have the option of declining the transition. 


Planned Evaluations 

The functionality described above has been 
developed at NASA Langley through several 
simulation prototypes, building up from the simple 
ID vehicle and environment used in the exploratory 
usability study, through planar operations in 2D, 
and currently in a light-aircraft simulation and 
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environment representative of the terminal area 
around a typical general aviation airport. So far, 
these prototypes have been implemented in a 
laboratory simulation environment on PC class 
computers. Presently, the functionality is being 
ported to the NASA Langley’s Visual/Motion 
Simulator. A study investigating the effects of 
aircraft motion, turbulence, and extraneous cockpit 
vibrations on the usability of the H-mode concept in 
general and on specific types of haptic interaction is 
to be conducted in this simulator within the next 6 
months. A second study is planned in the 
laboratory simulation in this same timeframe. This 
study will investigate the interactions between 
control automation level (i.e., augmented manual 
control; conventional, fully coupled automation; 
and an implementation of the H-mode), task 
performance, pilot workload, and situation 
awareness. While each automation level may 
perform best relative to a specific task element or 
performance measure, the objective is to assess 
whether the H-mode provides improved overall 
performance across the full range of task and 
measures. 

If these simulation studies provide positive 
results, it would be desirable to proceed to initial 
flight implementations. Such an implementation 
would be fairly straightforward on existing systems 
research aircraft such as the Bonanza described by 
Duerksen [18]. Implementations on unmanned/ 
remotely supervised aircraft would also be of high 
value and interest. 

Summary and Discussion 

The H-mode defines a specific, yet malleable 
human-machine interface design, associated 
human-machine interactions, and system 
architecture that is applicable to a wide range of 
vehicles and operational environments. By 
integrating management of the full range of control 
automation functions into a single consistent 
interface system and natural set of interaction skills, 
many of the fundamental human-machine 
interaction problems exhibited by conventional, 
highly automated vehicles may be reduced or 
eliminated. Also, by facilitating positive transfer of 
training, knowledge, and experience between 
operational domains and reducing the potential for 
confusion, we expect significant improvements in 


safety, effectiveness and training productivity for 
both users and developers. Additionally, unlike 
conventional automation implementations, the 
breadth and malleability of H-mode concept allows 
integration of future technologies without changing 
the fundamental system design and associated 
operator skills. Further, the concept provides a 
pathway by which emerging automation 
technologies supporting increasingly complex and 
safety-critical behaviors can be introduced and 
operationally evaluated while retaining effective 
human oversight. 

Much research and development remains 
before these claims may be realized, but the 
potential combined benefits of increased safety, 
ease of use, and performance make it a worthwhile 
task. NASA and the DLR are investigating 
comprehensive aerospace and ground vehicle 
applications respectively. At NASA, a simulated 
prototype for aircraft has been developed in a 
laboratory environment. This simulation is 
currently being ported to a motion-base simulator 
where the effects of motion on the concepts overall 
usability and design space will be evaluated. A 
second experiment exploring the interaction 
between automation level, workload, task 
performance, and situation awareness is also 
planned for the near future. Positive results from 
these experiments should establish the basic 
feasibility and impact of the H-mode concept in 
aircraft operations, and combined with the work at 
the DLR, will guide the future development of this 
promising concept. 

References 

[1] Billings, Charles, 1997, Aviation automation: 
The Search for a Human-Centered Approach. 
Mahwah, NJ: Lawrence Erlbaum Associates. 

[2] FAA Human Factors Team, 1996, “The 
Interfaces Between Flightcrews and Modem Flight 
Deck Systems”, Washington D. C., FAA 

[3] Wiener Earl, 1998, “Cockpit Automation”, In E. 
L. Wiener & D. C. Nagel (Eds.), Human Factors In 
Aviation , San Diego, CA: Academic Press, pp. 433- 
461 

[4] Sherry, Lance; Peter, Poison; Michael, Feary, 
2001, “Designing User-Interfaces for the Cockpit: 
Five Common Design Errors and how to Avoid 


12 



Them”, Society of Automotive Engineers, 02WAC- 
98, Warrendale, PA 

[5] NTSB, 1996, Washington, D. C., National 
Transportation Safety Board, www.ntsb . gov/ntsb/ 
GenPDF.asp?id=DCA96RA020&rpt=fa 

[6] NTSB, 1999, Washington, D. C., National 
Transportation Safety Board, 

www.ntsb . gov/publictn/2000/AAB000 1 .pdf 

[7] Andy Pasztor, 2006, “Airbus Planes to Use 
Computers In Crash- Avoidance Maneuvers”, Wall 
Street Journal, May 27 2006, New York, NY, Dow 
Jones and Company 

[8] Steering Committee for the Decadal Survey of 
Civil Aeronautics, 2006, “Decadal Survey of Civil 
Aeronautics: Foundation for the Future ”, 
Washington, D. C., The National Academies Press 

[9] Williams, Kevin, 2004, “A Summary of 
Unmanned Aircraft Accident/Incident Data: 

Human Factors Implications”, Oklahoma City, OK, 
Civil Aerospace Medical Institute, FAA 

[10] Minsky, Marvin; Push Singh; Aaron Sloman, 
2004, “The St. Thomas Common Sense 
Symposium: Designing Architectures for Human- 
Fevel Intelligence”, AI Magazine, summer 2004, 
Menlo Park, CA, AAAI Press 

[11] Flemisch, Frank, C. Adams, S. Conway, K. 
Goodrich, M. Palmer, P. Schutte, 2003, “The H- 
metaphor as a guideline for vehicle automation and 
interaction”, Technical Memorandum No. 
NASA/TM— 2003-212672, Hampton, VA, FaRC 

[12] German National Equestrian Federation, 2002, 
The Principles of Driving, Buckingham, UK, 
Kenilworth Press. 

[13] Schutte, Paul, Ken Goodrich, et al 2006, “The 
Naturalistic Flight Deck System: An Integrated 
System Concept for Improved Single-Pilot 
Operations”, Draft TM, Hampton, VA, LaRC 

[14] Flemisch, Frank; Ken Goodrich, Shelia 
Conway, 2005, “ITS At the crossroads of manually 
controlled and automated transport: The H- 
Metaphor and its first applications (progress update 
2005)”, Hannover Germany, Presented at The 5th 
European Congress and Exhibition on Intelligent 
Transport Systems and Services HIT 


[15] Salber, Daniel, Joelle Coutaz, 1993, “Applying 
the Wizard of Oz Technique to the Study of 
Multimodal Systems”; INTERCHI 93 Conference, 
New York, NY, pp 95-96. 

[16] Flemisch, Frank, J. Kelsch, A. Schieben, J. 
Schindler, 2006, “Stiicke des Puzzles 
hochautomatisiertes Fahren: H-Metapher und H- 
Mode“, 4. Workshop Fahrerassistenzsysteme; 
Fowenstein, Germany 

[17] Fambregts, Anthony, 2005, “Fundamentals of 
Fly-By-Wire Augmented Manual Control”, Society 
of Automotive Engineers, 05WAC-62, Warrendale, 
PA 

[18] Duerksen, Noel, 2003, “Advanced Flight 
Controls and Pilot Displays for General Aviation”, 
AIAA 2003-2647. 2003 AIAA/ICAS International 
Air and Space Symposium and Exposition, Dayton 
OH 

[19] Fiu, Dahai, Ken Goodrich, Bob Peak, 2006 
“Study of Synthetic Vision Systems (SVS) and 
Velocity- vector Based Command Augmentation 
System (V-CAS) on Pilot Performance”, Institute of 
Industrial Engineers Annual Conference and 
Exposition 2006, Orlando FL 

[20] Borst C., M. Mulder, M. M. van Paassen, J. A. 
Mulder, 2006 “Path-Oriented Control/Display 
Augmentation for Perspective Flight-Path 
Displays”, Journal of Guidance, Control, and 
Dynamics , Vol. 29, Reston, VA, AIAA 

[21] Burken, John, Peggy Williams-Hayes, John 
Kaneshige, Susan, Stachowiak, 2006, 
“Reconfigurable Control with Neural Network 
Augmentation for a Modified F-15 Aircraft”, 

NASA TM-2006-213678, DFRC, Edwards, CA 

[22] Arkin, Ronald 1998, Behavior-Based Robotics, 
Cambridge, Massachusetts, MIT Press 

25 th Digital Avionics Systems Conference 
October 15, 2006 


13 


